Teacher-student collaborative knowledge distillation for image classification

A single model usually cannot learn all the appropriate features with limited data, thus leading to poor performance when test data are used. To improve performance, we propose a teacher-student collaborative knowledge distillation (TSKD) method based on and self-distillation. The consists of two parts: learning in teacher network self-teaching student network. Learning allows use from Self-teaching is build multi-exit self-distillation provide deep as supervised information for training. In inference stage, ensembles vote classification results multiple sub-models experimental demonstrate superior our compared traditional self-distillation-based